Think Globally, Embed Locally --- Locally Linear Meta-embedding of Words

摘要

Distributed word embeddings have shown superior performances in numerousNatural Language Processing (NLP) tasks. However, their performances varysignificantly across different tasks, implying that the word embeddings learntby those methods capture complementary aspects of lexical semantics. Therefore,we believe that it is important to combine the existing word embeddings toproduce more accurate and complete \emph{meta-embeddings} of words. For thispurpose, we propose an unsupervised locally linear meta-embedding learningmethod that takes pre-trained word embeddings as the input, and produces moreaccurate meta embeddings. Unlike previously proposed meta-embedding learningmethods that learn a global projection over all words in a vocabulary, ourproposed method is sensitive to the differences in local neighbourhoods of theindividual source word embeddings. Moreover, we show that vector concatenation,a previously proposed highly competitive baseline approach for integrating wordembeddings, can be derived as a special case of the proposed method.Experimental results on semantic similarity, word analogy, relationclassification, and short-text classification tasks show that ourmeta-embeddings to significantly outperform prior methods in several benchmarkdatasets, establishing a new state of the art for meta-embeddings.